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^Parent ratings of young children s behavior are 
often used to assess social-emotional development. 
However, parents from different cultural and lan- 
guage backgrounds may vary in their (1) perceptions 
of the importance of different social-emotional indi- 
cators, (2) willingness to discuss particular behav- 
iors, and (3) ability to understand certain words 
and phrases. This brief presents findings from the 
Universal Preschool Child Outcomes Study, which 
examined a diverse group of Los Angeles preschool- 
ers. The study integrated three strategies — card 
sorts, focus groups, and cognitive interviewing — to 
look at how children s social-emotional behavior 
could be reliably and validly assessed through parent 
reports. Rather than identify one piloting strategy as 
more valid, we stress the importance of examining 
measures developed by and for clinicians and the 
implications of using these measures with diverse 
populations. 

Importance of Parent Ratings 

A child’s ability to develop relationships and con- 
nections with others, a sense of who he or she is in 
the world, and an ability to control impulses and 
emotions are at the core of social-emotional develop- 
ment. In early childhood, parent reports are one of 
the most common methods for assessing this type of 
development. Because parents observe their children 
in multiple settings, they can provide valuable insight 



ABOUT THE STUDY 



In spring 2007, Mathematica’s team embarked 
on the Universal Preschool Child Outcomes 
Study (UPCOS), a study of the Los Angeles 
Universal Preschool (LAUP) program for First 
5 LA. Parent interviews required a measure 
that provides a culturally sensitive assessment 
of children’s social-emotional competence, 
because Los Angeles is home to a culturally 
diverse population. 



into two components of social-emotional behavior: 

(1) social skills, and (2) problem behaviors. 

However, parents from different socioeconomic and 
cultural backgrounds vary in their perceptions about 
behaviors and their relative importance. For example, 
one study found differences in ratings related to a 
parent’s employment status, as well as preschooler’s 
gender and language level. Looking at how one item 
relates to other items on a scale, researchers find the 
reliability for parent reports is often lower than for 
teacher reports. More work is needed to examine the 
reliability and validity of parent ratings of children’s 
social-emotional development across various cultural 
groups. There is a need to examine how culturally 
and linguistically diverse parents (1) understand items 
relating to children’s social and emotional behaviors, 
and (2) perceive the importance of different social 
skills and problem behaviors. 

Clinical Tools, Diverse Audiences 

As part of the instrument design for UPCOS, we 
identified a battery of parent-reported items to 
assess children’s social-emotional development with 
a sample of parents of culturally and linguistically 
diverse preschoolers. We selected four commonly 






used measures that had been used with diverse 
populations and had established norms. We then 
conducted focus groups, card sorts, and cognitive 
interviewing, all described below, to shed light on 
how parents understand the items in these measures. 

There are several important facts to consider when 
selecting measures of social-emotional develop- 
ment. Most measures were developed by clinicians 
to be used with parents or teachers, not for use in 
large-scale surveys. They do not include definitions 
of terms or examples of behaviors, and the word- 
ing of some items may be difficult to interpret. For 
example, our review found that some items contain 
double negatives when used in combination with 
the response scales. Survey researchers often 
use the measures with parents who have varying 
levels of education and come from a variety of 
backgrounds (such as different countries of origin 
or language backgrounds). These individuals may 
have little understanding of clinical jargon that 
might be familiar to parents with more education. 

Getting Group Insights 

in May 2007, we conducted six focus groups with 
parents of preschool-age children from varying racial/ 
ethnic and linguistic backgrounds. Focus groups 
enable a small group to discuss a topic under the 
direction of a facilitator who guides the discussion. 
Individual focus groups consisted of parents who 
were Filipino (n=13), Korean (n=8), Chinese and 
Cambodian (n=10), African American (n=9), Latino 
Spanish-speaking monolingual (n=10), and Latino 
bilingual or English-speaking monolingual (n=7). We 
provided interpreters for members of the Filipino, 
Korean, and Chinese focus groups, as necessary. 

We conducted the Latino Spanish-speaking mono- 
lingual focus group in Spanish, and the other Latino 
focus group in English. The majority (63 percent) of 
parents participating in the focus groups had at least 
some college education. One-quarter were born in 
the United States. Most parents spoke English (51 
percent) and/or Spanish (23 percent) as the main 
language at home. 

The focus groups enabled parents from diverse racial/ 
ethnic and linguistic backgrounds to highlight and 
discuss social skills and problem behaviors that 
concerned them and were of most importance to 



SAMPLE FOCUS CROUP QUESTIONS 



“What does it mean to you when you say your 
child is well behaved?” 

“What social skills do you think going to pre- 
school will help your child have?” 

“Are there questions about your child’s behav- 
ior or well-being that you do not want to be 
asked about in an interview?” 



them. Parents also shared what they thought pre- 
schools should do to help their child’s development 
in these areas. All focus group discussions, includ- 
ing the card-sorting task, were audiorecorded. Then 
coders reviewed the tapes and highlighted themes 
and behaviors that were important to families. This 
process also identified common themes across parent 
responses in all focus groups. For example, “respect 
for adults” and “manners” were mentioned in several 
groups as important skills for children. We consid- 
ered these cross-cutting themes when selecting items 
and areas for discussion in subsequent cognitive 
interviews. 

Identifying Response Difficulty 

During the focus groups, participants took part in a 
card-sorting task. This involved looking at a group 
of index cards that each contained a single question 
about parent-reported measures of children’s social 
and emotional behaviors. We selected 5 1 questions 
from the standardized measures under consideration 
for the card-sorting task. Parents sorted each card into 
groups as follows: (1) easy to answer, (2) confus- 
ing and/or hard to answer, and (3) uncomfortable to 
answer, and put each group of cards into a labeled 
envelope. After items were sorted and envelopes 
collected, the focus group moderator asked parents to 
discuss the items that were confusing or made them 
uncomfortable. The purpose was to assess whether 
parents understood the intended meaning of the 
selected questions and whether they felt comfortable 
answering them. More broadly, the task examined 
the acceptability of different ways of asking about 
children’s behavior. 

After the focus groups, we tabulated responses to the 
card-sorting activity to identify differences among 
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parents in each of the focus groups in citing various 
items as easy, hard to understand, or uncomfortable to 
answer. Figure 1 shows the results. 

The findings suggested that we needed to adapt 
selected items for subsequent cognitive interviews and 
the final parent interview. Because the measures were 
copyrighted, we also had to think about whether the 
publisher would be willing to consider changes for use 
with our diverse sample. 

In the end, we selected the Preschool and Kindergar- 
ten Behavior Scale-2 (PKBS-2) for further piloting 
(Merrell 2003). We considered parents’ grouping 
of PKBS-2 items as easy to answer, hard to answer, 
and uncomfortable to answer, along with publisher- 
reported factor loadings of items, when selecting 
material for the cognitive interviews. We excluded 
items with comparatively low factor loadings (that is, 
a low relationship between the individual variable and 
the overall construct it was measuring) on subscales. 
We also excluded items that multiple parents cited as 
hard or uncomfortable to answer. 

As Figure 2 shows, parents from different groups 
viewed the sensitivity of the item, “Gives in or 
compromises with peers when necessary,” differently. 
While the item was intended to capture a positive 
trait, some parents felt that “giving in” was not posi- 
tive and therefore had difficulty responding to this 
item. This was especially true for Latino parents. As 



FIGURE 1 



Items Parents Considered Easy, Hard, or 
Uncomfortable to Answer, by Race/Ethnicity 
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a result, we excluded it from subsequent cognitive 
interviews and the final parent interview. 

Latino-English speaking parents and Latino-Spanish 
speaking parents differed sharply in their answers to 
a question on their child’s impulsiveness, as Figure 3 
shows. We explored the reason for divergence such as 
this in the cognitive interviews. If the divergence was 
the result of differential understanding of the item, it 
was modified or excluded. 

Thinking Aloud: Cognitive Interviews 

In June 2007, we conducted a series of cognitive inter- 
views with a different group of 53 parents to under- 
stand their process and rationale for answering items 
about their child’s social skills and problem behaviors. 
The cognitive pretests were limited to parents who 
spoke English or Spanish. The interviews included 
items about how cultural differences are addressed 
in preschool. Interviewers took extensive notes and 
revised drafts of the survey according to parents’ com- 
ments. Revised drafts were then used in the next round 
of interviews. 

The majority of parents participating in the cognitive 
interviews (60 percent) had at least some college edu- 
cation. Many were born in the United States (43 per- 
cent). In addition, most spoke English (66 percent) or 
Spanish (25 percent) as the main language at home. 
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FIGURE 3 




During the interviews, parents filled out a sample 
questionnaire containing demographics and the 
PKBS-2 social-emotional items. On the PKBS-2, 
parents rate social skills and problem behaviors on a 
4-point response scale from never to often. We used 
both English-only and Spanish-English bilingual 
versions of the questionnaire, so if a bilingual parent 
did not understand a term in English, he or she could 
refer to the Spanish translation. Then we asked par- 
ents questions similar to those posed during the focus 
groups. For example, we asked whether any of the 
PKBS-2 items were difficult to understand or other- 
wise confusing. We also asked them to explain what 
they thought individual items meant and to provide 
examples of observed behaviors that informed their 
responses to items. Interviewers probed items that 
parents rated at either extreme. 

From these discussions we identified a handful 
of PKBS-2 items in English and Spanish that had 
ambiguous meanings or that included Spanish words 
not commonly used in Los Angeles. We then added 



additional items to the questionnaires to test alterna- 
tive wordings and explored how parents interpreted 
them. Only a handful of items, usually involving 
technical or clinical language, needed wording 
or translation changes before the final pilot. For 
example, one item that posed difficulty for parents 
included the phrase “clings to parent or caregiver.” 
Some parents interpreted this as a positive behavior 
(offering “when my child is excited” as an example 
of when it occurs). This item was replaced with two 
items: “Holds onto a parent or caregiver when strang- 
ers are around,” and “Stays very close to a parent or 
caregiver for a long time in a new place.” 

The Meaning of Words 

We concluded that, before using established mea- 
sures in a study — especially measures developed for 
other purposes or audiences — it is critical to ensure 
that potential respondents understand the intended 
meaning of all items. Researchers cannot assume that 
respondents who speak English fully understand the 
concepts we are asking about in English — words have 
multiple meanings, and subtle differences in mean- 
ing can lead to misunderstanding of the question and 
skewed results. Clarifying these subtle differences in 
the understanding of a word or a phrase ensures that 
research provides the best measure for the population 
being studied. 

The full report on which this brief is based, “Los Angeles Univer- 
sal Preschool Programs, Children Served, and Children’s Progress 
in the Preschool Year: Final Report of the First 5 LA Universal 
Preschool Child Outcomes Study,” is available at www.first51a. 
org/research/UPCOS. Authors are John Love, Sally Atkins-Bumett, 
Cheri Vogel, Nikki Aikens, Yange Xue, Maricar Mabutas, Barbara 
Lepidus Carlson, Emily Sama Martin, Nora Paxton, Margaret 
Caspe, Susan Sprachman, and Kathy Sonnenfeld. For more 
information, contact senior fellow and survey researcher Susan 
Sprachman at ssprachman@mathematica-mpr.com. 
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